@ ACT Research & Policy 


TECHNICAL BRIEF 


MAY 2017 


Acknowledgement: The authors 
thank Krista Mattern and Emily Neff 
for their reviews and suggestions 
for improving this report. 


Evidence for Standard Setting: 
Probabilities of Success in “Benchmark” 
College Courses, by ACT Test Scores 


JEFF ALLEN, PHD, JUSTINE RADUNZEL, PHD, AND JOANN Moore, PHD 


Introduction and ACT ELA scores (Radunzel, Westrick, 
The ACT College Readiness Benchmarks Bassiri, & Li, 2017). Table 1 provides the 

are the ACT scores associated with a 50% ACT College Readiness Benchmarks and the 
chance of earning a B or higher grade in respective college courses used to establish 
selected first-year credit-bearing courses the Benchmarks. Students scoring at each 

at a typical postsecondary institution. The Benchmark have a 50% chance of earning a 
Benchmarks were established by linking B or higher grade in the respective course or 
ACT test scores with grades in first-year courses at a typical postsecondary institution, 
college courses from the same subject area. and approximately a 75% chance of earning 
Benchmarks were first established for ACT a C or higher grade. The purpose of this brief 
English, Mathematics, Reading, and Science is to present the full range of course success 
scores (Allen & Sconing, 2005; Allen, 2013) probabilities, by ACT test score, for each of the 
and have also been established for ACT college courses or course combinations listed 
STEM (Mattern, Radunzel, & Westrick, 2015; in Table 1. 


Radunzel, Mattern, Crouse, & Westrick, 2016) 


Table 1. ACT College Readiness Benchmarks 


ACT Test Score College Courses Benchmark 
English English Composition | 18 
Mathematics College Algebra 22 


American History, Other History, Psychology, Sociology, 


Renan Political Science, Economics 2 
Science Biology 23 
STEM' Calculus, Chemistry, Biology, Physics, Engineering 26 
ELA2 English Composition I, American History, Other History, 20 


Psychology, Sociology, Political Science, Economics 


’ The ACT STEM score is the rounded average of the ACT mathematics and science test scores. 
2 The ACT ELA score is the rounded average of the ACT English, reading, and writing scores. 
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Evidence for Standard Setting 


The ACT College Readiness Benchmarks 
can be used by states and districts as cut 
scores for performance levels indicating 
college readiness. However, rather than 
adopting the ACT College Readiness 
Benchmarks, states or districts may be 
interested in setting their own performance 
standards for the ACT test. Methods for 
setting performance standards (including 
standards for “proficient” and “college- 
ready”) require panelists to use their 
professional judgment to determine what 
level of performance is sufficient for 
meeting the standard. 


Traditional methods for establishing 
performance standards are based on 
content, requiring panelists to review 
entire tests or samples of test items. More 
recently, as greater emphasis has been 
placed on college and career readiness, 
empirically-based standard setting 
methods have become more prevalent, 
where the evidence upon which cut score 
determinations are made includes actual 
student success in college. Both methods 
require some level of professional 
judgment. In a content-based standard 
setting, the judgment typically relates 

to the knowledge and skills expected of 
a minimally college-ready student and 
the probability that such a student would 
answer a given test item correctly. For 

an empirically-based method such as 
that used to develop the ACT College 
Readiness Benchmarks, the judgments 
focus on three decisions: 


¢ What outcome reflects college 
readiness? The Benchmarks use first- 
year college course grades in some of 
the more commonly taken courses in 
the same subject areas as outcomes. 


¢ What constitutes success in each 
course? The Benchmarks use B or 
higher grades as the indicators of 
SUCCESS. 


¢ What probability level is indicative 
of readiness? The Benchmarks use 
a 50% chance of success as the 
threshold for readiness. 


Because judgment was exercised in 
developing the ACT College Readiness 
Benchmarks, it would be possible for a 
standard setting panel to recommend 

a different set of college readiness 

cut scores, even if they based their 
recommendation on the same set of data 
used to establish the Benchmarks. 


In some cases, multiple cut scores such 
as Basic, Proficient, and Advanced may 
be required. Multiple forms of evidence 
can be used during the standard setting 
process, which could include the empirical 
evidence used to develop the ACT College 
Readiness Benchmarks. The evidence 
supporting the Benchmarks includes the 
probabilities of different levels of course 
success (C or higher, B or higher, and 

A grades), by each ACT score point. 
These probabilities can be used to help 
standard setting panelists understand the 
relationship between potential cut scores 
and success in first-year college courses. 


In this brief, we present the course 
success probabilities, by ACT test score, 
for each of the college courses or course 
combinations listed in Table 1. While the 
ACT research reports documenting the 
development of ACT Benchmarks contain 
some information on course success 
probabilities, the full ranges of probabilities 
across the ACT score scales are not 
presented consistently in both graphical 
and tabular format. Moreover, the 
research report detailing the Benchmarks 
for English, mathematics, reading, and 
science (Allen, 2013) does not include 
the course success probabilities obtained 
after weighting the sample to make it 
representative of the population of ACT- 
tested college enrollees with respect to 
institution type and selectivity. In this brief, 


we present the weighted probabilities 
supporting all six ACT College Readiness 
Benchmarks. 


Course Success Probabilities 


On the pages that follow, the course 
success probabilities are presented 

in graphical and then tabular format. 
Probabilities are presented for three levels 
of success (C or higher, B or higher, and 
A grades). The probabilities represent 

the chance of success at the typical 
postsecondary institution. As described in 
the cited research reports, the probabilities 
are derived by fitting logistic regression 
models with intercepts and slopes specific 
to institutions and courses. The logistic 
regression model allows us to estimate 
probabilities at each ACT score, even 

for score regions with no or very few 
students. In each table, we provide the 
sample size at each score point so the 
reader can identify regions of the score 
distributions where extrapolations are 
being made. 


In summary, these tables and figures 

can be used by states and districts when 
considering local definitions of college 
readiness or for setting cut scores at 
other performance levels such as Basic or 
Advanced. ACT defines college readiness 
as having the knowledge and skills 
necessary to succeed in credit-bearing 
college courses without remediation. Tying 
performance on the ACT to predicted 
performance in college provides an 
empirical definition of college readiness 
that is based on actual student outcomes. 
Other data sources, including impact 

data at the state and national level and 
state-specific post-secondary enrollment, 
remediation, and longer-term college 
outcomes, can support the process 

of defining college readiness or other 
performance levels. 
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Figure 1. Probability of success in English Composition |, by ACT English score 
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Figure 2. Probability of success in College Algebra, by ACT mathematics score 


@ ACT Research & Policy Evidence for Standard Setting: Probabilities of Success in “Benchmark” College Courses, by ACT Test Scores 


Social Science Courses 
— -A =—Borhigher —=—-Cor higher 


1.00 


0.75 


0.50 


Probability of Success 


0 0 a 
123 45 6 7 8 9 1011 1213 1415 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 


ACT Reading Score 


T T T 


Figure 3. Probability of success in social science courses, by ACT reading score 
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Figure 4. Probability of success in Biology, by ACT science score 
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Calculus, Biology, Chemistry, Physics, Engineering 
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Figure 5. Probability of success in STEM courses, by ACT STEM score 


English Composition | and Social Science Courses 
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Figure 6. Probability of success in ELA courses, by ACT ELA score 
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Table 2. Probability of Success in English Composition |, by ACT English Score 


English Composition | Success Level 


ACT English Score N A B or higher C or higher 
1 0 0.010 0.098 0.449 
2 0 0.012 0.111 0.474 
3 1 0.014 0.126 0.500 
4 5 0.017 0.141 0.524 
5 9 0.020 0.162 0.545 
6 33 0.023 0.181 0.561 
7 109 0.028 0.200 0.585 
8 373 0.032 0.221 0.609 
9 498 0.038 0.244 0.625 

10 888 0.044 0.266 0.648 
11 1,234 0.051 0.295 0.670 
12 1,511 0.059 0.328 0.692 
13 2,023 0.067 0.356 0.716 
14 3,531 0.077 0.393 0.746 
15 5,489 0.091 0.425 0.757 
16 5,879 0.106 0.460 0.771 
17 5,858 0.122 0.496 0.790 
18 6,558 0.144 0.528 0.808 
19 8,466 0.165 0.559 0.819 
20 9,828 0.190 0.588 0.832 
21 9,499 0.220 0.618 0.847 
22 7,869 0.249 0.648 0.859 
23 5,659 0.283 0.675 0.871 
24 5,248 0.322 0.706 0.890 
25 4,839 0.363 0.730 0.899 
26 3,163 0.401 0.756 0.905 
27 2,067 0.448 0.790 0.912 
28 1,865 0.486 0.814 0.920 
29 1,358 0.523 0.834 0.928 
30 927 0.560 0.853 0.934 
31 628 0.596 0.870 0.941 
32 407 0.636 0.886 0.946 
33 297 0.681 0.897 0.951 
34 306 0.719 0.908 0.955 
35 138 0.754 0.919 0.959 
36 20 0.784 0.929 0.963 
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Table 3. Probability of Success in College Algebra, by ACT Mathematics Score 


ACT Mathematics College Algebra Success Level 
Score N A B or higher C or higher 

1 0 0.001 0.014 0.082 

2 0 0.002 0.018 0.095 

3 0 0.002 0.021 0.112 

4 0 0.003 0.026 0.129 

5 0 0.003 0.031 0.148 

6 0 0.004 0.038 0.171 

7 0 0.006 0.045 0.197 

8 1 0.007 0.055 0.224 

9 2 0.009 0.065 0.254 
10 4 0.012 0.078 0.286 
| 13 0.016 0.093 0.321 
12 65 0.020 0.111 0.346 
13 236 0.025 0.133 0.393 
14 803 0.033 0.158 0.434 
15 2,095 0.042 0.186 0.460 
16 4,757 0.053 0.218 0.513 
17 6,044 0.067 0.262 0.559 
18 5,978 0.086 0.305 0.596 
19 6,870 0.106 0.352 0.629 
20 6,467 0.134 0.404 0.663 
21 6,474 0.161 0.459 0.698 
22 5,670 0.196 0.507 0.726 
23 5,853 0;:235 0.552 0.755 
24 5,985 0.287 0.592 0.781 
25 4,232 0.340 0.639 0.802 
26 3,313 0.392 0.688 0.826 
27 2,204 0.451 0.732 0.848 
28 1,261 0.509 0.769 0.868 
29 877 0.575 0.801 0.885 
30 389 0.638 0.830 0.900 
31 306 0.697 0.859 0.914 
32 199 0.745 0.879 0.922 
33 159 0.785 0.900 0.935 
34 138 0.827 0.916 0.945 
35 55 0.860 0.931 0.953 
36 14 0.887 0.942 0.959 
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Table 4. Probability of Success in Social Science Courses, by ACT Reading Score 


Social Science Course Success Level 


ACT Reading Score N A B or higher C or higher 
1 0 0.011 0.060 0.243 
2 1 0.013 0.067 0.261 
3 | 0.015 0.077 0.280 
4 0 0.017 0.088 0.304 
5 10 0.019 0.100 0.329 
6 22 0.022 0.113 0.356 
7 45 0.026 0.126 0.384 
8 62 0.030 0.140 0.409 
9 127 0.035 0.156 0.436 

10 417 0.040 0.173 0.465 
ay 112 0.046 0.192 0.494 
12 2,504 0.052 0.216 0.521 
13 3,519 0.060 0.242 0.553 
14 5,799 0.071 0.266 0.581 
15 5,285 0.082 0.295 0.608 
16 7,362 0.093 0.319 0.633 
17 8,085 0.106 0.350 0.659 
18 6,972 0.122 0.378 0.685 
19 10,009 0.139 0.410 0.709 
20 11,775 0.159 0.441 0.725 
21 9,456 0.181 0.471 0.743 
22 8,680 0.207 0.505 0.758 
23 10,794 0.232 0.538 0.776 
24 8,856 0.259 0.572 0.795 
25 7,036 0.286 0.605 0.812 
26 6,818 0.313 0.634 0.832 
27 6,041 0.346 0.664 0.844 
28 5232 0.377 0.696 0.856 
29 4,452 0.409 0.724 0.870 
30 4,030 0.442 0.748 0.883 
31 3,453 0.479 0.774 0.894 
32 2:39/ 0.512 0.795 0.903 
33 2,209 0.550 0.815 0.914 
34 716 0.588 0.835 0.921 
35 980 0.622 0.852 0.928 
36 568 0.654 0.867 0.934 
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Table 5. Probability of Success in Biology, by ACT Science Score 


Biology Success Level 


ACT Science Score N A B or higher C or higher 
1 1 0.001 0.011 0.074 
2 0 0.002 0.013 0.086 
3 | 0.002 0.017 0.098 
4 0 0.002 0.020 0.116 
5 0 0.003 0.025 0.138 
6 3 0.004 0.031 0.157 
7 3 0.005 0.039 0.182 
8 21 0.006 0.048 0.206 
9 51 0.008 0.057 0.232 

10 149 0.010 0.066 0.264 
11 235 0.013 0.078 0.299 
12 340 0.016 0.093 0.337 
13 468 0.020 O113 0.380 
14 587 0.025 0.136 0.422 
15 826 0.032 0.162 0.470 
16 1,339 0.040 0.194 0.518 
17 1,832 0.051 0.227 0.566 
18 2,858 0.063 0.268 0.612 
19 3,770 0.080 0.307 0.652 
20 4,615 0.098 0.359 0.682 
21 4,345 0.119 0.408 0.713 
22 4,231 0.143 0.456 0.751 
23 3,787 0.175 0.507 0.785 
24 3,349 0.213 0.552 0.813 
25 2,585 0.255 0.598 0.837 
26 2,205 0.303 0.644 0.859 
27 1,226 0.356 0.695 0.876 
28 849 0.410 0.731 0.895 
29 569 0.473 0.771 0.907 
30 335 0.533 0.799 0.922 
31 345 0.593 0.834 0.934 
32 221 0.653 0.855 0.946 
33 210 0.703 0.880 0.952 
34 130 0.751 0.897 0.959 
35 111 0.792 0.915 0.966 
36 57 0.828 0.929 0.972 
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Table 6. Probability of Success in STEM Courses, by ACT STEM Score 


STEM Course Success Level 


ACT STEM Score N A B or higher C or higher 
1 0 0.001 0.010 0.048 
2 0 0.001 0.012 0.056 
3 0 0.002 0.014 0.064 
4 0 0.002 0.017 0.075 
5 0 0.002 0.020 0.086 
6 0 0.003 0.024 0.100 
7 0 0.004 0.029 0.116 
8 0 0.005 0.034 0.133 
9 0 0.006 0.041 0.152 

10 3 0.007 0.050 0.175 
ay 41 0.009 0.059 0.200 
12 50 0.011 0.071 0.229 
13 164 0.013 0.085 0.261 
14 356 0.016 0.101 0.295 
15 708 0.020 0.119 0.332 
16 1,312 0.026 0.139 0.372 
17 2,148 0.033 0.163 0.414 
18 3,287 0.041 0.189 0.457 
19 4,237 0.052 0.224 0.501 
20 4,815 0.065 0.259 0.544 
21 5,434 0.082 0:297 0.587 
22 6,245 0.102 0.346 0.630 
23 7,082 0.126 0.396 0.667 
24 8,123 0.154 0.442 0.706 
25 8,400 0.186 0.489 0.743 
26 8,007 0.227 0.544 0.777 
27 6,639 0.271 0.597 0.804 
28 5,068 0.319 0.640 0.826 
29 3,845 0.366 0.680 0.849 
30 2,965 0.423 0.719 0.872 
31 2,206 0.485 0.754 0.887 
32 1,554 0.547 0.791 0.901 
33 1,036 0.602 0.820 0.915 
34 742 0.652 0.844 0.927 
35 387 0.699 0.868 0.938 
36 95 0.745 0.889 0.946 
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Table 7. Probability of Success in ELA Courses, by ACT ELA Score 


ELA Course Success Level 


ACT ELA Score N A B or higher C or higher 
1 0 0.009 0.053 0.229 
2 0 0.010 0.061 0.250 
3 0 0.012 0.071 0.268 
4 0 0.015 0.080 0.286 
5 5 0.017 0.093 0.310 
6 12 0.020 0.108 0.334 
7 62 0.024 0.121 0.361 
8 107 0.029 0.138 0.389 
9 210 0.034 0.156 0.417 

10 399 0.040 0.177 0.450 
11 862 0.048 0.200 0.484 
12 1,514 0.057 0.225 0.514 
13 2,790 0.066 0.251 0.545 
14 4,563 0.078 0.279 0.578 
15 6,909 0.091 0.314 0.612 
16 9,751 0.106 0.350 0.649 
17 12,198 0.124 0.388 0.677 
18 14,957 0.144 0.427 0.706 
19 17,103 0.165 0.466 0.730 
20 18,279 0.191 0.506 0.755 
21 18,398 0:222 0.547 0.778 
22 17,559 0.252 0.585 0.804 
23 15,862 0.289 0.622 0.824 
24 13,449 0.322 0.662 0.843 
25 11,268 0.360 0.697 0.857 
26 9,139 0.402 0.734 0.874 
oT 7,244 0.446 0.766 0.888 
28 5,432 0.490 0.793 0.899 
29 3,889 0.532 0.820 0.910 
30 2,614 0.575 0.842 0.920 
31 1,743 0.615 0.859 0.930 
32 1,033 0.652 0.878 0.938 
33 633 0.689 0.894 0.946 
34 252 0.725 0.909 0.952 
35 37 0.758 0.922 0.957 
36 2 0.783 0.932 0.961 


Note. The sample used to develop the ACT ELA Benchmark was comprised of students who had taken the former ACT Writing test. Students’ ACT ELA scores were estimated 
by calculating the rounded average of the ACT English, reading, and concorded writing scores. 
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